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APPLICATION  OF  BIORTHONGONAL 
FILTER  FUNCTIONS  TO  PATTERN  RECOGNITION 
AND  FEATURE  EXTRACTION 


INTRODUCTION 


It  is  typical  of  many  problems  of  pattern  recognition  that  the  number  of  possible 
environmental  states  to  be  considered  may  be  infinite,  but  that  the  number  of  solutions 
or  classifications  that  can  be  assigned  remains  finite.  A  set  of  measurements  is  performed 
on  an  environmental  system,  and  by  using  the  measurements,  a  single  category  is  as¬ 
signed  to  this  pattern  of  measurements.  Thus,  the  system  is  recognized  as  belonging  to 
only  one  class. 


In  cases  where  terrain  is  examined,  the  environmental  system  must  sometimes  be 
viewed  as  a  collection  of  objects  belonging  to  several  different  classes.  Again,  in  cases 
where  remotely  sensed  data  is  used,  the  pattern  of  measurements  obtained  can  consist 
of  a  superposition  of  patterns  from  objects  belonging  to  different  classes.  In  these  two 
cases,  one  must  determine  the  relative  proportion  of  the  different  components.  In  other 
cases,  the  object  to  be  identified,  may  be  viewed  against  any  one  of  a  number  of  different 
backgrounds  that  produce  strong  interferences  in  measurements  made  on  the  system  and 
it  becomes  desirable  to  separate  or  filter  the  object  parameters  from  the  background. 


In  the  first  two  cases,  methods  more  general  than  the  normally  used  simple  pattern 
recognition  scheme  are  required.  In  the  third,  at  least  some  additional  processing  is  in¬ 
dicated.  It  will  be  shown  that  mathematical  expansion  in  biorthogonal  sets  of  functions 
can  be  used  to  advantage  in  solving  these  problems. 


The  methods  to  be  developed  are  general  and  the  approach  is  phenomenological 
since  they  can  be  applied  irrespective  of  the  type  of  measurements  performed  on  the 
system.  For  example,  the  spatial  frequency  spectrum  may  be  used,  or  some  functions 
that  depend  on  it;  or  the  electromagnetic  spectrum  may  be  used,  or  some  integrated 
responses  for  wide  spectral  bands.  As  an  alternative,  other  types  of  measurements  or 
combinations  of  mixed  types  of  measurements  may  be  used. 


In  most  pattern  classification  schemes,  the  “classing”  of  an  object  depends  on  its 
correlation  with  patterns  of  an  idealized  reference  pattern  set  or  alphabet.  With  the 
method  to  be  introduced  here,  a  biorthogonal  filter  function  is  constructed  corresponding 
to  each  member  of  the  reference  set.  The  filter  function  has  the  (biorthogonal)  property 
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that  it  has  zero  correlation  with  all  members  of  the  reference  set  except  the  one  with 
which  it  corresponds.  This  property  has  the  effect  of  suppressing  cross  correlation  with 
unwanted  patterns  for  standard  pattern  recognition  problems  as  well  as  for  the  ones  to 
be  emphasized  here.  The  method  is  also  adaptive  in  the  sense  that  when  a  new  reference 
pattern  is  added  to  the  alphabet,  the  new  filter  function  can  be  constructed  and  the  old 
ones  revised  to  yield  a  new  enlarged  set  of  filter  functions  without  starting  the  com¬ 
putation  from  the  beginning. 


A  QUALITATIVE  DESCRIPTION  OF  THE  METHOD 


A  set  of  three  functions  or  reference  patterns  that  may  characterize  three  different 
objects  is  shown  in  figure  1 .  A  superposition  of  these  same  functions  is  given  schemat¬ 
ically  in  figure  2.  The  problem  is  to  determine  which  components  are  present  in  the 
superposition  and  in  what  amounts.  In  a  real  case,  the  number  of  reference  functions 
might  be  expected  to  be  much  greater  than  three,  and  the  superposition  may  contain  a 
small  number  of  them,  e.g.  two  or  three.  Some  of  the  reference  patterns  could  consist  of 
typical  background  noise  patterns  that  need  to  be  suppressed  in  the  process  of  system 
identification. 


The  method  for  constructing  the  filter  functions  is  demonstrated  schematically 
by  means  of  a  different  example  shown  in  figure  3.  Three  reference  patterns  are  shown 
in  the  upper  part  of  the  diagram  as  A,  B,  and  C.  Their  corresponding  biorthogonal 
filter  functions  are  given  below  them  by  A',  B',  and  C',  respectively.  They  have  the 
property  that  if  any  unprimed  function  is  multiplied  by  a  primed  function  and  inte¬ 
grated,  the  result  will  be  zero  except  for  the  case  where  the  filter  function  corresponds 
with  the  reference  pattern.  It  is  significant  that  each  filter  function  can  be  represented 
as  a  linear  combination  of  the  given  reference  functions  as  shown  by  the  equations  at 
the  bottom  of  figure  3. 
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Figure  1 .  Characteristic  Patterns  for  Three  Objects 


Figure  2,  Linear  Superposition  of  the  Reference  Patterns 


Construction  of 


MATHEMATICAL  DERIVATION 


The  set  reference  patterns  is  assumed  to  be  given  by  a  set  of  N  linearly  in¬ 
dependent  functions  f;(x),  where  /  =  1,  2 . N.  It  is  possible  to  construct  a 

second  set  of  functions  0*(x),  where  k  =  1,  2, . N,  such  that  they  have 

the  property 


fba  f/x)  <MX)  dx  =  8.k 


(1) 


where  5/Jt  is  the  Kronecker  delta.  Sets  of  functions  <t>k(x)  obeying  equation  (1) 
are  said  to  be  biorthogonal  (normalization  will  be  understood  as  well)  to  the  first 
set  f.(x)  over  the  interval  [a,  b] . 


For  a  finite  number  of  reference  functions,  it  will  usually  be  possible  to  con¬ 
struct  many  different  sets  of  biorthogonal  filter  functions.  However,  if  the  form 
of  the  filter  functions  is  limited  to  linear  superpositions  of  the  reference  functions 


yv 

#*(*)  =  £  «/*  f/x)  (2) 

i~i 

The  symmetric  matrix  J5  is  defined  in  terms  of  its  components  by 

f„ -  //,<*>  f„  (*>  dx  (3) 

and  it  will  possess  an  inverse  if  the  functions  f.(x)  form  a  linearly  independent  set 
and  the  determinant  of  J3  is  consequently  non-zero. 


If  equation  2  is  multiplied  by  fffl(x)  and  integrated  from  a  to  b  and  if  equations 
1  and  3  are  used,  then 


fm(x)4>j(x)dx 


EaH  / *  f«<x>f/(x)dx 


(4) 
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(5) 


r 


N 

5.  =  'Ta.,0. 

jm  /I  rlm 

1=1 


It  is  found  that  the  matrix  a.  is  the  inverse  of  JL  The  computation  of  the  components 
of  ^  is  all  that  is  required  to  find  the  biorthogonal  filter  functions  of  the  required 
form  as  expressed  in  equation  2. 


The  fact  that  a  reciprocal  relationship  exists  between  the  reference  functions 
can  be  readily  demonstrated  by  forming  the  symmetrical  autocorrelation  coefficient 
matrix  for  the  functions  0;(x), 


-b  LL  J 

f  0/x)  0fc(x)dx  =  Y,  %  /  f,(x)  4>k(\)  dx 


(6) 


1=1 


£  V,* 

1=1 


(7) 


a., 

Ik 


(8) 


where  use  has  been  made  of  equations  2  and  1 .  and  by  forming  the  sum 


£  P,k  <MX)  =  £  £  P,k  a*/f/(x) 


(9) 


k  =  l 


k  =  1  1=1 


~  ^  h  ff 

1=1  11  ‘ 


(X) 


(10) 


=  f(x) 


(M) 
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where  equations  2  and  5  have  been  used.  To  emphasize  their  character,  the  reciprocal 
relationships  of  equations  2  and  1 1  are  given  in  more  explicit  terms  below: 


N 

<t>kw  =  £  f/x) 

/=/ 


N 

fkw  =  £  ¥x) 

j°i 


<*(*)  f/8  d£ 


(12) 


(13) 


Since  every  member  of  either  set  of  functions  is  expressible  as  a  linear  combination  of 
the  members  of  the  other  set,  either  set  may  be  used  as  a  basis  to  span  the  same  function 
space. 
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FILTERING  OF  SIGNATURES 


Suppose  the  function  n(\)  consists  of  a  linear  superposition  of  patterns  f.(x) 
for  a  known  set  of  features,  then 


M(x) 


E 

i=l 


(14) 


If  the  biorthogonal  set  of  functions  $  (x)  is  known,  the  presence  and  relative 
amounts  of  each  component  pattern  is  readily  found. 


b  N  b 

f  n(\)  dx  =  £  a.  J  f;(x)  <t>k(x)  dx 


(15) 


i=i 


N 


=  E  a.  6.. 

i-i  /  Ik 

i=i 


(16) 


=  a. 


(17) 


It  should  be  emphasized  that  once  the  Filter  function  <t>kM  is  known,  the  pre¬ 
sence  or  absence  and  the  amount  of  the  k-th  component  can  be  found  without  deter¬ 
mining  the  presence  or  amounts  of  the  other  possible  components.  This  is  of  value  when 
searching  for  a  particular  feature  in  a  background  consisting  of  other  reference  patterns 
where  it  is  not  desirable  to  perform  computations  for  all  of  the  components  present. 


The  equations  to  be  presented  next  are  relevant  if  the  correlation  coefficients  of 
the  superposition  with  the  set  of  reference  patterns  is  known.  Their  derivation  is  straight 
forward  and  follows  from  the  reciprocal  properties  of  biorthogonal  sets  of  functions. 


M(x) 


N 


-  E 

k  =  l 


b* 


(18) 
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(19) 


f%(x)fm(x)dx  =  £b  k  f\(x)fm(x)dx 
•*  a  k  =  j  J  a 


N 


-  L  \  s,„ 


(20) 


=  b 


(21) 


The  relationship  between  the  a’s  and  b’s  is  determined  by 


N 


f  $j(x)  n(x)  dx  =  2  b*  /  0/x)^(x)dx 

J  a  iTz.  *  +  a 


k  =  1 


(22) 


N 

h  =  £  \aki 

k  =  l 


(23) 


or 


N 


b«=  £  M 


Im 


1=1 


(24) 


since  the  matrix  JL  is  the  inverse  of  j&. 
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APPROXIMATION  IN  THE  MEAN 


The  best  fit  of  the  function  p(x)  in  a  least-square  sense  results  in  the  same  co¬ 
efficients  as  those  given  by  equation  21,  and  the  proof  of  this  follows  in  an  analogous 
fashion  to  that  used  for  orthogonal  functions.  The  function 


N 

I  =  f  [  ju  (x)  —  £  y,  f,(x)  1 

J  a  i=i 


dx 


(25) 


is  to  be  minimized  by  adjusting  the  coefficients  7  . 


31 


fb  * 

=  0  =  2  f  [  #i(x)  -  £  7.  f.(x)  1  f  (x) 

“  l=l 


dx 


(26) 


/b  N  b 

fi(x)f  (x)dx  =  ^  7  f  f  (x)f  (x)dx  (27) 

0  1=  i  J  0 


N 


=  £  y, 

1=1 


(28) 


By  multiplying  both  sides  by  amn  and  summing  over  m. 


N 


f  n(x)  T'  a  f  (x)  dx  = 

Ja  L-i  mn  ’ 


m  =1 


N  N 

y  y  7.  a  a 

*-/  ‘l  rlm  m  n 


1=1  m  =1 


(29) 


/%(x)«„(x)  dx  =  £  7<$;„ 


i=i 


a  =  7 

n  'n 


(30) 

(31) 
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it  is  found  that  the  coefficients  7  and  _a_  are  the  same. 


The  quantity 


S 


f  l  *x)  -  Tan  f  (x)  ]2  dx 
J  a  n*l 


(32) 


may  be  used  as  a  measure  of  the  fit  of  v(x)  by  the  set  of  functions  fn(x)  *n  cases  where 
Kx)  is  a  pattern  from  the  real  world  environment.  A  representation  in  terms  of  reference 
functions  is  not  usually  exact,  and  it  is  important  to  know  the  quality  of  the  fit  and  the 
improvement  accomplished  by  changing  or  introducing  additional  reference  (basis) 
functions. 


N 


s  =  f  [  Kx)  |2  dx  -  2  £  a,  f  rtx)  f„(x) 


dx 


N  N  b 

+  ^  ^  ^ a  ^ 


(x)  frn(x)  dx 


(33) 


=  fm  l  "<X>  l2  dx  -  2  £*„*>„  +  E  £a«  am  Vmn  (34) 

a  n  =  l  n  =  Jm  =  l 


=  f  f  "<x)  i‘’dx  - 2  £a„  b„  +  £  a„  b„ 


(35) 


n  =  /  n  -  / 


=  f  [  p(x)  ] 2  dx  -  0 

J  a 


(36) 


N 

a  =  £  a  b 

n  n 

n  =  l 


(37) 
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ADAPTIVE  LEARNING 


The  ability  of  the  system  to  recognize  patterns  may  be  increased  by  the  addition 
of  new  reference  patterns.  This  requires  the  computation  of  a  new  set  of  filter  functions. 
In  this  section  it  will  be  shown  that  this  may  be  accomplished  by  using  the  old  set  with¬ 
out  recomputing  and  starting  from  the  beginning. 


Suppose  the  biorthonormal  set  [  0n(x):  k  =  1 ,  N  - 1  ]  has  been  found  corres¬ 
ponding  to  the  set  [  f*(x):  k  =  1,  N  - 1  ]  and  then  the  latter  set  is  enlarged  to 
[  ffc(x):  k  =  1,  N]  by  the  addition  of  fN(x).  A  new  set  of  biorthogonal  filter  func¬ 
tions  I  0^(x):  k  =  l,  N]  will  result.  The  old  coefficients  &lk  where  /,  k^N-1 
will  remain  unchanged,  but  the  new  coefficients  &lN  and  &NN  must  be  computed 
and  appended.  Also,  a  new  set  of  coefficients  cTfc/;  k,  l  ^  N  must  be  found  to  replace 
the  old  ones  akl;  k,  l  ^  N  -  1 .  However,  use  can  be  made  of  the  old  coefficients 
akl  to  reduce  the  labor  involved  in  finding  akl. 


The  new  and  old  filter  functions  are  given  by 


f/<x> 


(40) 


and 


n-1 


(41) 


respectively. 


N-l 


E  01  k  l  ”, Im 


1=1 


+  a  f  f/v(x)0m(x)dx  (43) 

J  a 


+  “kN  //*<*>  *„<X>dX 


(44) 


Also, 


N-l 


Ja^(x)0m(x)  dx  =  £  <*w„  /fl^(x)f„(x)dx  (45) 


N-l 

y  a  5. 

m  n  K  n 


n  =  / 


(46) 


m* 


(47) 


Combining  equations  44  and  47  yields 


a,  =  a  .  -  a.  „  f  f.,(x)0  (x)dx 
fcm  m  k  kN  J  „  N  ' 

*  a 


(48) 


Next,  formulae  for  the  coefficients  of  the  type  akN  and  aNm  will  be  found. 


N-l 


/  Vx>*m<x)dx  =  E  f  Vx)f„(x)dx 


(49) 


N-l 

y  a  5V 

4— i  i»g  wn 

n  =  1 


(50) 


(51) 
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Also 


f*  <j>N(x)  dx  =  £  aNI  f  f,(x)  <t>m  (x)  dx  (52) 

J  a  I-]  J  a 

=  E  %lb,m  +  «NN  /\(X>^m(x)dX  (53) 

By  combining  equations  51  and  53 

0  =  E  %,b,m  +  “mm  j\(x>*«(x>dx  ($4> 

1=1  a 

%m  -  <55> 

and  by  inserting  equation  55  into  48,  the  coefficients  of  type  clkN  and  aNm  are 
eliminated. 

=  +  *W  j\(x>^<x>dx>  (  f'fvW^Wdx)  (56) 

*  a  *  a 

For  convenience  of  notation,  let 


N-l 


N-l 


K  =  //*(*)  *M(*)dx  =  J/*(x)f„(x>dx  =  £  (57) 


n  =  l 


n  =  1 


Then,  equation  56  becomes 


Skm  amk  +  N  \ 


(58) 
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For  the  new  set  of  filter  functions,  the  normalization  condition  for  0 
dated  with  the  biorthogonality  is 


1  =  /VA,(x)fyv(x)dx 


JV 

=  22  &NI  f  ^v(x)^(x)dx 

/=/ 


*  -V 


N-l 


22  +  ^NN01, 

1=1 


■JViV 


yv-/ 


where  the  substitution 


22  fyv/  “/V/V  \  +  &NN  °^N N 

1=1 


aM  ~  ~  &NN  \ 


has  been  taken  from  equations  55  and  57  above. 


Equation  62  can  now  be  solved  for  &NN- 


1 


NN 


1 


N-l 


N-l  N-l 


^NN  ~  22  ^Nl\  fyv/V  ~  22  22  ^ Niaim  ®Nm 

1=1  1=1  m  =1 


1 


asso- 


(59) 


(60) 


(61) 


(62) 


(63) 


(64) 
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r 


The  formulae  needed  for  the  revision  of  the  filter  functions  are  all  derived.  The  steps 
in  the  computational  procedure  are  given  below: 


1.  /?..  remain  unchanged  for  /,  k  =  1,  N  -  1) 
J* 


2.  Compute  $Nj  = 

■■  0.N  for  j  =  1,  N  from 

ii 

N-l 

Jb  fN(x)  f.(x)  dx 
a 

3.  Compute  X;  = 

E 

a.  for  /  = 

lm  ^ Nm 

m=l 

4.  Calculate  5 Nff 

1 

N-l 

@NN  ~  E  $Nl\ 

1=1 

5.  Compute  a  { 

= 

aNN  \  ^°r  *  ~  ^ 

6.  Compute  akm 

= 

oi,  +  a..,,  X  X, 

km  NN  m  k 

or 


a,  =  a,  a„.  X  for  k,  m  -  l,  N  -  1. 
km  km  Nk  m 


Where  possible,  use  the  symmetry  of  a  to  fill  out  its  values. 


The  procedure  given  here  amounts  to  a  learning  process  where  the  old  filter  func¬ 
tions  are  modified  (retrained)  to  ignore  the  new  pattern  being  added  to  the  set  of  re¬ 
ference  functions. 


19 


GENERAL  REMARKS 


In  practical  implementation,  the  integrals  will  often  be  replaced  by  summation  over 
some  finite  number  of  points.  The  function  space  (with  an  infinite  number  of  compon¬ 
ents)  is  thereby  reduced  to  a  finite  dimensional  vector  space.  To  retain  linear  independ¬ 
ence  of  the  reference  patterns,  the  number  of  components  must  be  at  least  as  great  as  the 
number  of  reference  patterns.  By  way  of  contrast,  no  such  condition  is  necessary  (al¬ 
though  it  may  hold  accidentally)  in  most  normal  pattern  recognition  procedures.  The  gain 
for  fulfilling  this  condition  is  the  ability  to  decompose  superpositions  of  patterns  into 
their  component  parts. 


Even  if  the  filter  functions  were  used  with  the  usual  pattern  recognition  pro¬ 
cedures,  the  separability  of  many  patterns  should  be  increased.  In  addition,  the  weight 
of  any  single  component  can  be  obtained  by  itself  by  using  its  appropriate  filter  function. 
It  should  also  be  mentioned  that  typical  backgrounds  and  interferences  can  be  included 
in  the  reference  pattern  set,  and  their  usual  tendency  to  interfere  can  be  somewhat 
suppressed  in  this  manner. 


CONCLUSION 


The  concept  of  biorthogonality  has  been  developed  into  a  method  of  pattern 
recognition  that  can  be  applied  to  a  wide  variety  of  data  types.  This  treatment  leads  to 
mathematical  methods  for  constructing  filter  functions  that  give  perfect  response  to 
one  member  of  the  set  of  given  functions  while  giving  zero  response  to  all  of  the  others. 
Furthermore,  this  response  can  be  found  by  probing  the  environmental  state  with  only 
the  one  filter  function. 


The  mathematical  system  is  capable  of  learning  new  patterns  and  revising  the 
old  filter  functions  so  that  they  have  no  response  to  the  new  patterns.  In  contrast  to 
standard  pattern  recognition  methods  where  one  classification  is  made,  this  method 
permits  the  decomposition  of  a  superposition  of  many  patterns  into  its  component  parts. 
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